This invention relates to a method and system for reproducing sounds in an interactive virtual world environment.
"Virtual reality" is the experience of living or playing in a computer-generated environment which models a three-dimensional ("3-D") virtual space (i.e., a virtual world). In virtual reality systems, a viewer typically dons a set of goggle-mounted video screens or some other form of head-mounted display ("HMD"), and an audio headset, to block out the real world. Typically, the HMD is a conventional pixel (picture element) based, raster scan video display device. The viewer might also be provided with a simulated laser gun, a vehicle with accelerator and brake pedals, or some other device coupled to the computer system to enable the viewer to move about in, interact with or otherwise influence objects and characters in the virtual world. Sounds emanating from the virtual world (such as lasers firing or vehicles accelerating) are reproduced by speakers in the viewer's headset or by external speakers.
On the HMD the viewer sees images of virtual world scenes that are generated from a 3-D model of the virtual world by a computer graphic system. The scenes are displayed to the viewer as they appear from a specific position and orientation in the virtual world, called the "viewpoint" or "eyepoint." Usually, the viewer is given some degree of control over the position and orientation of the viewpoint, thus allowing the viewer to see different images from a plurality of different viewpoints.
By enabling the viewer to change the position and/or orientation of his or her viewpoint, the computer graphic system can create the illusion of the viewer "travelling" through the virtual world and looking in all directions. Depending on the capabilities and programming of the system, the viewer might be allowed to "travel" without restriction above, below and around a scene, as well as into or through structures, as though the viewer could fly or had other capabilities. The system might also be designed to constrain the motion of the viewpoint in various ways to achieve realism. For example, the viewer might only be allowed to position and orient the viewpoint no closer than six feet from the ground to simulate a view of the virtual world from the vantage of a person standing on the ground in the world. Alternatively or in addition, the viewpoint might be constrained from passing through the image of a solid surface (such as the wall of a building)--just as it is typically impossible in the real world to walk through the wall of a building. Also, the viewer might be constrained to move along a defined path, as though traveling on a train, such that a series of events can be enacted as the viewer's train passes by predetermined positions in the virtual world.
Virtual reality systems have developed from traditional military and commercial airline flight simulators, and military tank simulators, in which computer graphic systems render a simulated, 3-D world from the perspective of a person looking out of an aircraft cockpit window or tank turret (i.e., the system's "viewpoint"). The world created by such simulators typically includes static structures and terrain (e.g., an airport with runways and buildings situated in a world including lakes, rivers and mountains), and moving objects (e.g., flying aircraft, land vehicles and clouds). The images of the simulated world displayed on the "windows" of the cockpit or turret continually change--as might occur in the real world--in response to changes in position and attitude of the aircraft or vehicle being "flown" or "driven" by the person participating in the simulation.
Virtual reality systems have applicability in the entertainment industry. Computer games and arcade machines presenting a virtual reality experience as a form of entertainment can provide a viewer with a high level of enjoyment. Virtual reality systems can immerse a viewer in a realistic world, or a highly fantasized or magical one where even the laws of physics have been skewed, to produce an entertainment experience available by no other means.
Prior 3-D virtual reality systems, however, have suffered from a number of drawbacks that limit the virtual experience in ways that are particularly detrimental to the use of such systems to produce high-quality entertainment. One such drawback is the inability of such systems to immerse the viewer in a richly detailed 3-D auditory illusion which complements the visual illusion of the 3-D graphic virtual world.
Providing such a richly detailed 3-D auditory illusion requires a method and system capable of generating a large number of sounds and delivering those sounds in a three-dimensional space surrounding the viewer. Because the viewer can change position and orientation in the virtual world relative to other virtual world objects, including objects representing sound sources in the virtual world, a method and system which is capable of adjusting a richly detailed auditory illusion in real time as a function of the viewer's position and orientation in the virtual world would be preferred, particularly one that can give a viewer the impression that sound sources are localized as in the real world.
Some 3-D computer graphics systems that are known today incorporate a real time spatial sound processor for manipulating select monophonic audio signals so that they are perceived by a viewer wearing stereo earphones as sounds originating from sources located at predetermined positions in a 3-D virtual space. In these systems, the perceived positions of the sound sources are coordinated with the positions of graphic objects in the virtual world. The viewer thus perceives that the sounds are coming from the objects at their positions in the virtual world. Yet these systems provide only a relatively simple auditory environment. They lack mechanisms for managing a complex 3-D auditory presentation involving viewer movement, changing virtual world scenes and viewer interaction such as would be desirable for producing high quality entertainment.
The current per-channel cost of real time spatial sound processors makes them impractically expensive for reproducing a large number of separately localized sounds simultaneously. However, a realistic 3-D audio illusion can be provided in a virtual reality system without requiring that each sound which contributes to the audio environment of the virtual world be specifically localized to a particular 3-D position by a spatial sound processor. A few key sounds can be localized while other less important sources in the environment are reproduced using less expensive non-localized techniques. By appropriately selecting the sounds that are localized, the lack of localization for other sounds in the environment can be disguised. For instance, the playing of a non-localized sound of a door opening simultaneously with a localized sound of a door knob turning may lead a viewer to believe that both sounds are localized, especially if an effect such as reverberation is added to the non-localized sound to provide dimension.
Even so, complex virtual reality systems may make unpredictable demands on a complementary sound system. Although the events that take place in a virtual world created by a virtual reality system are generally each pre-planned and scripted, the scripting may allow spontaneous changes in the arrangement of those events, or in the events themselves, to be induced by movements and actions of the viewer. Thus, the number and nature of the sounds required for a particular virtual world scene may vary depending on viewer movement and interaction or some other not-fully predictable factor. For such systems a flexible scheduling mechanism which functions automatically to control the generation and delivery of sounds in accordance with current virtual world events may be required.
The demands made of the sound system may occasionally exceed its capacity to generate and deliver sounds in a desired manner, or at all, as may happen for example if a viewer's actions cause an unexpectedly large number or an unexpected combination of virtual world events to take place simultaneously, or if cost restraints preclude the system designer from incorporating optimum resources in the sound system. For example, by design or accident a virtual world scene might have more key sounds to be localized simultaneously than the number of available spatial sound processor channels. Likewise, the total number of sounds to be generated simultaneously may exceed the number of sound generator channels available. Some of these sounds may be more important to the virtual world scene than others. Thus it may be required that the resources of the sound system be allocated in a prioritized manner to increase the likelihood that an important sound will not be denied a sound generator channel or an appropriate sound delivery channel (e.g., a channel of a spatial sound processor).
In view of the foregoing, it would be desirable to be able to enhance a computer-generated, graphic virtual world environment with a complementary auditory environment provided by a virtual world sound system that implements a flexible and automatic resource scheduling method and system capable of interacting with the viewer.
It would also be desirable to be able to provide in such a sound system a prioritized resource allocation method and system to increase the likelihood that an important sound is allocated appropriate sound system resources in accordance with a desired manner of playback.